IN THE CLAIMS : 

Please amend Claims 1-4, 8-10, 18-23, 28, 29, and 37-39 as follows: 

1 . (Currently Amended) A simulator apparatus with which an operator plays a 
simulation with virtual obj e ct(s) in mixed reality space including a virtual space and real 
space in which a real obiect(s) is placed , said simulator comprising: 

a viewpoint detection unit adapted to detect the location/postur e 
position/orientation of a viewpoint of the operator; 

an inputting unit adapted to input a real space image corresponding to the 
location/postu r e position/orientation of a the viewpoint of the operator; 

a geometric information acquisition unit adapted to recogniz e acquire 
geometric information of the real object(s) placed in the real space ; 

a rule memory adapted to store rules for controlling the action of the virtual 

object(s); 

a computation unit adapted to determine the next action of the virtual object(s) 
by referring to said rule memory based on a relation among the location/posture 
position/orientation of a the viewpoint of the operator, location(s) position^) of the virtual 
object(s) and the geometric information of the real object(s); and 

a p re sentation virtual space generation unit adapted to generate at l e ast on e 
image of th e virtual obj c ct(s) a virtual space image on the basis of the location/posture 
position/orientation of the virtual object(s) after the determined action and the 
location/ p ostu re position/orientation of the viewpoint position of the operato r, and to 



-2- 



r e pres e nt th e mixed r e ality spac e to the operator by superimposing the imagc(s) of the 
virtual objectfs) on the operato r 's vi e w of the r eal space. ; and 

a mixed reality image generation unit adapted to generate a mixed reality space 
image by superimposing or overlaying the virtual space image on the real space image. 

wherein said rul e s cont r ol th e action of th e virtual obj e ct(s) on th e basis of an 
obj e ctiv e and a relativ e location relationship between the virtual objcct(s) and th e r e al 
object(s) that is represented by the geom e tric information. 

2. (Currently Amended) The apparatus according to claim 1, whe re in said 
pr e s e ntation unit further compris e s 

wherein said inputting unit captures an imag e -captu r ing unit adapt e d to captu re 
real space images of said operator's view of re al space imag e s of said player's view of the 
real spaced;]],, 

an imag e g e n e ration unit adapted to generate mix e d r e ality images rep re s e nting 
th e mix e d reality spac e by superimposing or overlaying said imag c (s) of th e virtual 
obj e ct(s) on said r eal space imag e s; and 

and the apparatus further comprises a video see-through type display that the 
operator wears wherein said the mixed reality images are displayed. 

3. (Currently Amended) The apparatus according to claim 1, wh er ein said 
p re sentation unit further compris e s comprising an optical see-through type display that the 
operator wears wherein said virtual object imag e (s) are space image is displayed. 
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4. (Currently Amended) The apparatus according to claim 1, further 
comprising, comprising: 

a status detector that detects a status of the o per ato r ; operator, 
wherein said computation unit determines a next action of the virtual object in 
accordance with the rule stored in said rule memory and in correspondence with the 
location/posture position/orientation of the real object and/or the status of the operator, and 
computes a location/postu re position/orientation of the virtual object after the determined 
action. 

Claims 5-7 (Canceled) 

8. (Currently Amended) The apparatus according to claim 1, wherein the real 
object(s) include other operators who operate said simulator apparatus, and the anothe r 
other operators share a single mixed reality space with the operator. 

9. (Currently Amended) The apparatus according to claim 1, wherein the real 
object is an object which is fixed in position in the real space , and 

said geometric information acquisition unit comprises: 

a predetermined memory for pre-storing location position information and 

shape information of the real object; and 

a reading unit that reads out the location position information and shape 

information of the real object from said predetermined memory as needed. 



10. (Currently Amended) The apparatus according to claim 1, wherein the real 
object is an object which is movable but does not deform, and 

said geometric information acquisition unit comprises: 

a predetermined memory for pre-storing shape information of the real object; 

a location/postur e position/orientation sensor for detecting a location/posture 
position/orientation of the real object; and 

a setting unit that sets a region the real object is expected to occupy in the 
mixed real reality space in accordance with the detected location/postur e 
position/orientation of the real object. 



Claims 11-17 (Canceled) 



18. (Currently Amended) The apparatus according to claim 1, wherein said 
viewpoint detection unit detects a location/postur e position/orientation of the head of the ■ 
operator, and said apparatus further comprises; 

a detector that detects a location/posture position/orientation of a hand of the 
operator; and 

a recognition unit adapted to recognize a relative location position of the hand 
of the operator with respect to the head as a command on the basis of an output from said 
detector. 



19. (Currently Amended) The apparatus according to claim 1, wherein said 
presentation virtual space generation unit comprises: 
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an alignment unit that aligns the location/posture position/orientation of the 
real object to the location/postur e position/orientation of the virtual object after movement; 

a generation unit that generates an image of the virtual object after alignment in 
correspondence with an occlusion relationship; and 

a head-mounted display device. 

20. (Currently Amended) An image processing method for a simulator 
apparatus with which an operator plays a game witli virtual obj c ci(s) in a mixed reality 
space in which a real objectfs) is placed, comprising: 

a viewpoint detection step for detecting the location/postu r e 
position/orientation of a viewpoint of the operator; 

an inputting step for inputting a real space image corresponding to the 
location/postur e position/orientation of a the viewpoint of the operator; 

a geometric information acquisition step for acquiring geometric information 
of the real object(s) placed in the real space ; 

a computation step for determining the next action of the virtual object(s) by 
referring to the rul e m e mory rules for controlling the action of the virtual object(s) based 
on a relation among the location/posture position/orientation of a the viewpoint of the 
operator, location(s) position(s) of the virtual object(s) and the geometric information of 
the real object(s); and 

a p re sentation virtual space generation step for generating at least one image of 
the virtual object(s) a virtual space image on the basis of the location/posture 
position/orientation of the virtual object(s) after the determined action and the 
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location/ p ostu r e position/orientation of the viewpoint position of the operator , and for 
representing th e mixed reality spac e to the operator by supe r imposing th e imagc(s) of 
virtual objcct(s) on th e o p e r ator's view of th e real space, ; and 

a mixed reality image generation step for generating a mixed reality space 
image by superimposing or overlaying the virtual space image on the real space image. 

wher e in said rules cont r ol th e action of th e virtual object(s) on the basis of an 
objective and a r elativ e location r elationship between the virtual objcct(s) and th e r e al 
object(s) that is re p re s e nted by th e geom e tric information. 

21. (Currently Amended) The method according to claim 20, wherein th e 
op e rato r w e ars a video see-through type display, and said pr e s e ntation st e p furth er 
comp r ising, 

wherein said inputting imag e -capturing step captures for capturing real space 
images of said operator's view of the real space[[;]] a 

image g e neration step for g e n e rating mixed r e ality images repres e nting of the 
mixed r e ality space by superimposing or ov e rlaying said imag e (s) of virtual object(s) on 
said real space images and fo r displaying said mixed re ality images on th e display 

and the simulator apparatus comprises a video see-through type display that the 
operator wears wherein the mixed reality images are displayed . 

22. (Currently Amended) The method according claim to 20, wherein the 
operator wears an optical see-through type display and said p re s e ntation step representing 
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the mixed reality s p ace to th e operator by displaying the image(s) of virtual object(s) space 
image is displayed on the display. 

23. (Currently Amended) The method according to claim 20, further 
comprising, comprising: 

a status detecting step for detecting a status of the op er ato r ; operator, 
wherein said computation step determines a next action of the virtual object in 
accordance with the rule stored in said a rule memory and in correspondence with the 
location/postur e position/orientation of the real object and/or the status of the operator, and 
computing computes a location/posture position/orientation of the virtual object after the 
determined action. 

Claims 24-26 (Canceled) 

27. (Previously Presented) The method according to claim 20, wherein the 
real object(s) include other operators who operate the simulator apparatus, and wherein the 
other operators share a single mixed reality space with the operator. 

28. (Currently Amended) The method according to claim 20, wherein the real 
object is an object which is fixed in position in the real space , and 

the geometric information acquisition step includes the steps of: 
pre-storing location position information and shape information of the real 
object in a predetermined memory; and 
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reading out the location position information and shape information of the real 
object from the predetermined memory as needed. 

29. (Currently Amended) The method according to claim 20, wherein the real 
object is an object which is movable but does not deform, and the geometric information 
acquisition step includes the steps of: 

pre-storing shape information of the real object in a predetermined memory; 

detecting a location/posture position/orientation of the real object by a 
location/postur e position/orientation sensor; and 

setting a region the real object is expected to occupy in the mixed real reality 
space in accordance with the detected location/posture position/orientation of the real 
object. 

Claims 30-36 (Canceled) 

37. (Currently Amended) The method according to claim 20, wherein the 
viewpoint detection step includes the step of detecting a location/posture 
position/orientation of the head of the operator, and 

said method further comprises: 

the a detection step of detecting a location/ p ostu re position/orientation of a 
hand of the operator; and 

the a step of recognizing a relative location position of the hand of the operator 
with respect to the head as a command on the basis of an output in the detection step. 
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38. (Currently Amended) The method according to claim 20, wherein the 
pr e sentation virtual space generation step includes the steps of: 

aligning the location/posture position/orientation of the real object to the 
location/postur e position/orientation of the virtual object after movement; and 

generating an image of the virtual object after alignment in correspondence 
with an occlusion relationship. 

39. (Currently Amended) A storage medium which stores a program of an 
image processing method for a simulator apparatus with which an operator plays a 
simulation with virtual objcct(s) in a mixed reality space including a virtual space and real 
space in which a real obiect(s) is placed, comprising: 

a viewpoint detection program step for detecting the location/posture 
position/orientation of a viewpoint of the operator; 

an inputting program step for inputting a real space image corresponding to the 
location/postur e position/orientation of a the viewpoint of the operator; 

a geometric information acquisition program step for acquiring geometric 
information of the real object(s) placed in the real space : 

a computation program step for determining the next action of the virtual 
object(s) by referring to rules for controlling the action of the virtual object(s) based on a 
relation among the location/posture position/orientation of a the viewpoint of the operator, 
location(s) positions) of the virtual object(s) and the geometric information of the real 
object(s); and 



- 10- 



a presentation virtual space generation program step for generating at least one 
imag e of the virtual object a virtual space image on the basis of the location/postu r e 
position/orientation of the virtual object(s) after the action and the location/postu r e 
position/orientation of the viewpoint position of the operator , and for repr e senting the 
mixed reality space to the operator by superimposing the imagc(s) of the virtual objcct(s) 
on th e operator's view of the real spacc,; _and 

a mixed reality image generation program step for generating a mixed reality 
space image by superimposing or overlaying the virtual space image on the real space 
image. 

wh e rein said rules control the action of th e virtual object(s) on the basis of an 
obj e ctiv e and a relativ e location r e lationship between the virtual object(s) and the r eal 
obj c ct(s) that is represented by the geometric information. 
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